Skip to content

Encrypted DM reader, batch profiles, liked tweets, more human-like delays#7

Open
jackordman wants to merge 4 commits intonirholas:mainfrom
jackordman:encrypted-dm-reader
Open

Encrypted DM reader, batch profiles, liked tweets, more human-like delays#7
jackordman wants to merge 4 commits intonirholas:mainfrom
jackordman:encrypted-dm-reader

Conversation

@jackordman
Copy link
Copy Markdown

@jackordman jackordman commented Mar 12, 2026 •
edited
Loading

Summary

Four commits covering encrypted DM support, new scraping capabilities, anti-detection improvements, and bug fixes.

Encrypted DM reader (src/scrapers/twitter/dm.puppeteer.js)

  • New Puppeteer-based scraper for encrypted DMs — handles X's passcode gate (both inline and /i/chat/pin/recovery redirect), conversation list scraping, message extraction with sender detection, and media detection
  • MCP tools: x_read_dms, x_export_dms, x_get_conversations
  • CLI commands: xactions dms list, dms read <username>, dms export with interactive passcode prompt and optional save
  • Sender detection via background color heuristics (dark mode) with horizontal alignment fallback
  • skipNavigation optimization avoids redundant /messages navigation in export loops
  • Handles both /i/chat (new Tailwind UI) and /messages (legacy) DOM structures

Batch profile scraping (x_get_profiles)

  • New MCP tool for fetching multiple profiles in a single call
  • Human-like delays between navigations to avoid robotic browse patterns

Liked tweets scraping (scrapeLikedTweets)

  • New scrapeLikedTweets() scraper with optional username (auto-resolves from config or sidebar)
  • x_get_likes MCP tool wired to use it as single source of truth
  • Return shape standardized to { likedTweets, count, username }
  • Removed from xeepyTools, removed redundant double-wrapping in server handler

Human-like delays and anti-detection

  • Replace uniform random delays (1-3s) with log-normal distribution (2-7s) plus ~8% distraction spikes (8-20s) across scrapers and MCP local-tools
  • Deduplicated randomDelay in local-tools.js — now imported from scrapers module

Auth and reliability fixes

  • checkAuth() post-navigation guard for bookmarks and likes scrapers
  • Auto-login in MCP ensureBrowser() when XACTIONS_SESSION_COOKIE is set
  • Fix browser hang on auth failure with try/finally in unified scrape()
  • Resolve and save username on xactions login, clear on logout

Restored missing client files

  • Restore parsers.js, search.js, trends.js, TwoFactorAuth.js, DEFAULT_FIELD_TOGGLES — were deleted but still imported, breaking MCP server startup

Test plan

  • x_get_conversations — returns conversations with names + handles
  • x_read_dms — returns messages with sender detection, text, time, media
  • x_export_dms — loops multiple conversations via skipNavigation
  • x_get_likes — returns wrapped object, no double-wrapping
  • CLI dms list/read/export --help parses correctly
  • All modules load: dm.puppeteer.js, local-tools.js, server.js, CLI
  • Existing unit tests pass (221/222, 1 pre-existing failure in HTTP DM sendDM test)

@jackordman
feat: human-like delays, scrapeLikedTweets, auth checks, MCP auto-login
c09805b 
- Replace uniform random delays (1-3s) with log-normal distribution (2-7s)
  plus ~8% distraction spikes (8-20s) in scrapers and MCP local-tools
- Add scrapeLikedTweets() to scrape a user's liked tweets page, with
  optional username (auto-resolves from config or page nav sidebar)
- Save resolved username to ~/.xactions/config.json for future use
- Wire x_get_likes MCP tool to use scrapeLikedTweets (single source of truth)
- Add checkAuth() post-navigation guard for bookmarks and likes scrapers
- Fix browser hang on auth failure with try/finally in unified scrape()
- Auto-login in MCP ensureBrowser() when XACTIONS_SESSION_COOKIE is set
- Resolve username on xactions login and clear on logout
- Restore missing client files (parsers, search, trends, TwoFactorAuth)
  that were deleted but still imported, breaking MCP server startup
- Add DEFAULT_FIELD_TOGGLES back to graphqlQueries.js
- Update CLAUDE.md with sequential-request rule and delay documentation
@jackordman jackordman requested a review from nirholas as a code owner March 12, 2026 03:20
@vercel
Copy link
Copy Markdown

vercel bot commented Mar 12, 2026

@jackordman is attempting to deploy a commit to the aryllyraaryl-6220's projects Team on Vercel.

A member of the Team first needs to authorize it.

@jackordman jackordman force-pushed the encrypted-dm-reader branch from 03f7cde to 9140aaa Compare March 12, 2026 03:24
@jackordman jackordman changed the title Add encrypted DM reader — MCP tools + CLI Encrypted DM reader, batch profiles, liked tweets, human-like delays Mar 12, 2026
@jackordman jackordman changed the title Encrypted DM reader, batch profiles, liked tweets, human-like delays Encrypted DM reader, batch profiles, liked tweets, more human-like delays Mar 12, 2026
@jackordman
feat: add x_get_profiles batch MCP tool
6aa3c3d 
Batch profile scraping with human-like delays between navigations,
avoiding robotic browse patterns when fetching multiple profiles.
@jackordman
fix: x_get_likes return shape and routing
a11b25d 
Move x_get_likes out of xeepyTools (Python shim) since it's handled
directly by local-tools. Wrap return value in { likedTweets, count,
username } at the local-tools layer and remove redundant double-wrapping
in the MCP server handler.
@jackordman
feat: integrate encrypted DM reader into MCP server and CLI
ee90cf7 
Add Puppeteer-based encrypted DM scraper module with passcode gate
handling, conversation list scraping, and message extraction with
sender detection. Wire into MCP tools (x_read_dms, x_export_dms,
x_get_conversations) and CLI (dms list/read/export).

- skipNavigation option avoids redundant /messages nav in export loops
- Defensive back-navigation when split-pane sidebar collapses
- Fuzzy matching constrained to display name elements only
- Transparent background skip in sender color detection
- Document CLAUDE.md stealth infrastructure requirement
@jackordman jackordman force-pushed the encrypted-dm-reader branch from 9140aaa to ee90cf7 Compare March 12, 2026 03:54
@nirholas
Copy link
Copy Markdown
Owner

nirholas commented Mar 12, 2026

Thank you! Will get to these as soon as i can. Looks great from what I saw at first glance. I appreciate it.

@nirholas
Copy link
Copy Markdown
Owner

nirholas commented Mar 30, 2026

Hey @jackordman, thanks for this — solid work overall. I'd like to take the safe improvements now and split the DM features into a follow-up PR. Here's what I need:

Please split this into two PRs:

PR 1 — Bug fixes & improvements (merge now):

Log-normal randomDelay() distribution
checkAuth() post-navigation guard
scrapeProfile error handling for suspended/missing accounts
scrapeBookmarks auth check
scrapeLikedTweets scraper + x_get_likes routing fix (remove from xeepyTools, use proper scraper)
try/finally browser cleanup in scrape()
DEFAULT_FIELD_TOGGLES export
CLI scrape actions fix (likes, media added)

PR 2 — DM features (needs changes before merge):

Encrypted DM scraper, MCP tools, CLI commands
x_get_profiles batch tool
Before resubmitting, please:
Rebase onto main — parsers.js, search.js, and trends.js already exist on main, so the "restore" commits conflict. Drop those files from your branch.
Drop TwoFactorAuth.js — it's a stub that always throws, let's add it when it's implemented.
Scope the auto-login in ensureBrowser() — injecting the cookie on every browser creation affects all 50+ MCP tools. Move the loginWithCookie() call into the DM-specific functions instead.
Remove the config file writes from scrapeLikedTweets — a scrape function shouldn't silently write to ~/.xactions/config.json.
CLI login/logout username resolution can stay in PR 2 but separate from the passcode storage logic.

No issues with the DM scraper itself — the sender detection heuristic and passcode gate handling look good for a v1.
Just want to keep the blast radius small per PR. Appreciate the contribution! 🙏

@jackordman
Copy link
Copy Markdown
Author

jackordman commented Mar 30, 2026

@nirholas No problem! Just some stuff I put together while messing around. I'll make these changes when I find some time after work this week

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@nirholas nirholas Awaiting requested review from nirholas nirholas is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@jackordman @nirholas